Skip to content

[DTLS 1.3] Refine Handshake_Layer and Channel_IO for (Post-)Handshake Epoch Enforcement - #6074

Open
reneme wants to merge 2 commits into
randombit:masterfrom
Rohde-Schwarz:dtls13/refine_handshake_layer
Open

reneme wants to merge 2 commits into
randombit:masterfrom
Rohde-Schwarz:dtls13/refine_handshake_layer

Conversation

@reneme

@reneme reneme commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Pull Request Dependencies

Description

This is a fairly broad refactoring of the Handshake_Layer, and by extension also the Channel_IO, to better re-use parts of the handshake message parsing and validation logic. It became necessary to properly validate some handshake message protocol invariants.

Namely:

  • Record-boundary checks for (post-)handshake messages
    Certain handshake messages must be aligned with a record boundary. Namely those messages that cause a traffic key epoch change: ClientHello, ServerHello (+ HRR, HVR), Finished, and KeyUpdate.
  • (Post-)Handshake messages that follow a key change must be encrypted with the new epoch
    That is trivially true for TLS, but DTLS provides a window between sending KeyUpdate and waiting for its ACK where this invariant can be broken. RFC 9147 does not explicitly require this invariant, but BoringSSL (and BoGo) does. We also implement it for compatibility and because it harmonizes the epoch-ordering guarantees of post-handshake messages of TLS and DTLS
  • Handshake records holding fragments of the same message must not be interleaved by other record types
    This is an invariant that applies to TLS only. In DTLS that cannot be guaranteed due to out-of-order delivery of datagrams. But until now, the respective check would have required a workaround once DTLS comes around (namely: DTLS_Handshake_Layer::has_pending_data() always returning false)

Further changes:

  • Channel_IO: restructure maybe_send_opportunistic_key_update() so that it decides once whether to send a single KeyUpdate and whether to request a peer update. As a side effect, a peer update is no longer requested when the
    peer has already half-closed its side of the connection. A test is added for that case.
  • Channel_IO: make notify_closed_for_reading() virtual, add the virtual notify_closed_for_writing() (called by Channel_Impl_13 once close_notify was sent) and let notify_shutdown() invoke both. validate() is no longer const so that implementations may keep state while inspecting passed-through records.

That should be the final PR before #5000 can land. Hopefully.

@reneme
reneme requested a review from randombit October 30, 2026 15:55
@reneme reneme self-assigned this Oct 30, 2026
Copilot AI balanced review requested due to automatic review settings October 30, 2026 15:55

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@reneme reneme mentioned this pull request Oct 30, 2026
@atreiber94
atreiber94 force-pushed the dtls13/refine_handshake_layer branch 3 times, most recently from a1e9902 to 51b24da Compare November 5, 2026 16:41
@reneme

reneme commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator Author

CI fails because this is currently not based on #6104. Edit: fixed.

@reneme
reneme force-pushed the dtls13/refine_handshake_layer branch 4 times, most recently from 4a5d592 to e14d297 Compare November 10, 2026 12:49
In preparation for DTLS 1.3 (randombit#5000), introduce all flavor-dependent
logic that will be in the TLS base files: every place where the base 1.3
implementation must behave differently for DTLS now branches on
TLS_Flavor, and the polymorphic layer interfaces (Channel_IO,
Handshake_Layer, Record_Layer, Cipher_State) expose the hooks that the
DTLS 1.3 implementation will override. The DTLS 1.3 implementation
itself is not part of this change: creating a DTLS 1.3 client still
throws, and DTLS 1.3 remains disabled via Policy::allow_dtls13()
defaulting to false.

Concretely:
* Channel_Impl_13 takes a Connection_Side and TLS_Flavor; Client and
  Server select the 1.3 implementation for is_tls_13_or_later()
  versions and pass the flavor down. Cipher_State::create() and
  Channel_IO::create() gain (still unreachable) DTLS branches that
  throw Not_Implemented until the DTLS 1.3 module lands.
* Client/Server Hello 1.3 handle the DTLS legacy_version, ciphersuite
  lists and supported_versions, skip the middlebox compatibility mode
  and the session ID echo for DTLS; Hello_Retry_Request may carry a
  Cookie for the DTLS stateless cookie exchange; Encrypted_Extensions
  negotiates DTLS-SRTP (RFC 5764) and the negotiated SRTP profile is
  stored in the session.
* Channel_IO grows no-op notification hooks for the DTLS ACK and
  retransmission logic (protocol version committed, complete/final
  flight received); Record_Layer and Handshake_Layer report ingestion
  outcomes and support epoch-specific record protection as needed for
  DTLS.
* KeyUpdate handling: Channel_IO::handle_key_update() becomes virtual
  and subsumes the schedule_read_epoch_pruning() hook, which is
  removed. The DTLS implementation overrides it to reject KeyUpdates
  beyond the epoch limit (RFC 9147 8.) and to schedule pruning of
  superseded read epochs itself; TLS simply updates its read keys.
* Downgrade handling: the client preserves epoch-0 sequence numbers for
  the 1.2 implementation; random_signals_downgrade() moves to the
  Server_Hello base class. Handling of Hello_Verify_Request as a
  downgrade trigger to DTLS 1.2 is factored out into a subsequent
  commit, as it belongs to the DTLS 1.3 implementation itself.
* PSK binder calculation handles the DTLS legacy_cookie field and the
  PSK importer uses the "dtls13" label prefix for DTLS 1.3.
* Policy gains allow_dtls13() (default: false).

The handling of ACK records (RFC 9147 7.) together with the
dtls_maximum_queued_acknowledgements() policy knob is likewise factored
out into a subsequent commit, as the DTLS 1.3 implementation is their
only consumer.

Co-Authored-By: René Meusel <rene.meusel@rohde-schwarz.com>
@reneme
reneme force-pushed the dtls13/refine_handshake_layer branch from e14d297 to a8f50ec Compare November 10, 2026 13:48
Prepares the shared TLS 1.3 base layers for the DTLS 1.3 implementation
module. All changes are in TLS base files and behave identically for
TLS, with one exception noted below.

* Handshake_Layer: replace the per-flavor virtual next_message() and
  next_post_handshake_message() by non-virtual implementations that are
  built on a single hook, fetch_next_message_buffer_view(). The flavors
  merely provide the next message (including its epoch, if any) and the
  base class does the policy size check, parsing, transcript update and
  the RFC 9846 5.1 "no key change within a record" validation. The
  handshake message type validation no longer distinguishes handshake and
  post-handshake phases; the parsers reject unexpected types as before.
  The validation of excess handshake data moves from Channel_IO into the
  handshake layer, hence the BoGo error mapping is adapted.
* Channel_IO: introduce can_send_post_handshake_messages() in place of
  can_send_key_update(); Channel_Impl_13 and Server_Impl_13 refuse to send
  KeyUpdates and NewSessionTickets when it is false (DTLS may have to wait
  for the acknowledgement of a pending KeyUpdate). The opportunistic
  KeyUpdate logic in send() is folded into
  maybe_send_opportunistic_key_update(). Add the validate() hook for
  records that Channel_IO merely passes through, so TLS can reject the
  interleaving of handshake and other records while DTLS may tolerate it.
* Generalize variant_append_t to append multiple alternatives.
* Channel_IO: restructure maybe_send_opportunistic_key_update() so that it
  decides once whether to send a single KeyUpdate and whether to request a
  peer update. As a side effect, a peer update is no longer requested when the
  peer has already half-closed its side of the connection.
* Channel_IO: make notify_closed_for_reading() virtual, add the virtual
  notify_closed_for_writing() (called by Channel_Impl_13 once close_notify was
  sent) and let notify_shutdown() invoke both. validate() is no longer const
  so that implementations may keep state while inspecting passed-through
  records.

Co-Authored-By: René Meusel <rene.meusel@rohde-schwarz.com>
@reneme
reneme force-pushed the dtls13/refine_handshake_layer branch from a8f50ec to 3cc60b3 Compare November 10, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants